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I. INTRODUCTION 

The Curriculum Guideline Implementation 
Evaluation project was initiated in response to the 
need of the Ministry of Education of Ontario to assess its 
efforts in guiding curriculum development in the public 
and separate schools. The Ministry seeks to accomplish 
this objective by producing for each discipline documents 
called Curriculum Guidelines. With varying degrees , each 
guideline is intended to provide philosophies withdn which 
disciplines should be taught, possible teaching topics, 
resource materials, etc. Because different committees are 
responsible for producing and disseminating each guideline, 
the methods of content, writing styles, production and 
dissemination, etc. vary for each guideline. 

The intention of this project was to design a 
system which would permit the Ministry to examine and compare: 

a) Guideline distribution among th<^. ten regions in 
the Province ; 

b) Comprehension by educational personnel of the content 
of each guideline ; 

c) The degree of incorporation of guidelines into the 
curriculum ; 

d) Educational personnel perceptions of the impact of 
guidelinco in the classroom. 

The Work described herein was performed by the SDL Systems 
Research Group under contract for the Ministry of Education 
^ of Ontario. 
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Because of the diverse nature of the approximately 
130 guidelines, the ten geographic divisions (regions) in 
the province, the personnel involved in distribution and 
receipt of guidelines, it was necessary to design data 
collection instruments to permit an in-depth examination 
of each of thesse variables. 

As a result , three personnel level s were ident i- 
fied as being of particular importance with respect to guide- 
lines. These include Ministry personnel. Board personnel 
and school personnel. Within each of these, key personnel 
types were identi f ied as playing a part icularly maj or ro le 
in implementing guidelines. These were: 

Ministry Personnel: Ministry Program Consultants 
Board Personnel: Directors and Superintendents of 

School Boards 5 Board Program 
Consultants 
• School Personnel: Principals 

Teachers 

II. FIELD TEST 

In May of 1973 an extensive field test was conducted 
to evaluate the remaining alternatives for each phase of 
the overall system . These alternat ives were concerned 
primarily with the selection of a suitable data collection 
instrument and techniques of optimally utilizing the instru- 
ment in collecting information from educational personnel in 
this province. 

Because of the number of personnel type s to be 
accessed, it vjas necessary to investigate separately the 
effect of response rate on each. Compounding this personnel 
variable was the need to investigate regional differences 



for each personnel level. Finally, the field-test was 
required to consider the effect on response rate of gather- 
ing information on different guideline types. Table 1 
presents a summary of the different data collection instru- 
ments and the number distx^'ibuted to each personnel level. 

The questionnaires which were field-tested were 
preliminary versions of the one in Appendix A, A different 
role-related questionnaire was administered to each personnel 
level. 

The essay instrument consisted of an open ended 
version (for each personnel level) of the questionnaires. 

The distribution strategies compared were mailing 
the instrument di rect ly to the respondent versus ma i ling 
it to a superior of the respondent and instructing the 
superior to ensure the return of the instr^ument? ^nd personal 
interviews . 

Each of the cells in Table 1 was further broken 
down to ensure both a regional and guideline distribution 
for each. Data v;as gathered from five regions for seven 
guidelines. While strictly speaking the field test consisted 
ofa4x5x5x7 (instrument x personnel x region x 
guideline) design, practical considerations had to be imposed 
to limit each cell size. In addition, the primary objective 
fo?r conducting the field test was to obtain an intuitive 
guide concerning optimization of each possible approach to 
collecting data from the different personnel. 

The results of the field-test indicated that 
optimal response and utility was obtained by mailing 
questionnaires to the superiors of the respondents. No 
differences in this result were noted for either region or 
guideline type. 



TABLE 1 

PLAN FOR INSTRUMENT FIELD-TESTING 





Questionnaire 


Essay 




PERSONNEL 


Personally 
Administered 


Mailed and 
Distr^ibuted 


Mailed 

Direct 


Mailed 
Direct 


TOTAL 


Teachers 




mo 


178 




318 


Principals 






30 




30 


School Board 
Consultants 




20 


20 


20 


60 


Directors of Board 
of Education 




10 


10 




20 


Ministry Program 
Consultants 


5 




24 


10 


39 


TOTAL 


5 


• 170 


262 

i 


30 


467 
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Ill- PRIMARY EVALUATION 



On the basis of the field test, an evaluation 
was conducted throughoux the province. The objective of 
this evaluation was the creation of a primary data base 
and computer-based information system to be used to permit 
future evaluation data to be collected, added and compared. 
The primary data base consists of evaluation data on 
eighteen guidelines for all regions in the province. Table 
2 presents a list of each of the guidelines selected for 
the primary evaluation. 

The results of the survey were analys ed for each 
guideline. Appendix B of this report consists of typical 
reports generated by the system. The analyses shown in 
the Appendix consist of data for all guidelines combined 
and for all regions combined. 

Of the approximately 21,0 00 questionnaires mailed 
in this evaluation the overall response rate for all personnel 
levels was 50,2%, A regional breakdown of response rates is 
presented in Table 3, The results for each personnel level 
surveyed are presented in this regional breakdown. 

Appendix C consists of a regional breakdown of 
response for each guideline evaluated for each personnel 
level surveyed. 

Appendix D consists of the regional response for 
each guideline evaluated considered for English and French, 

Appendix E consists of a regional breakdov;n of 
response by guideline. 



GUIDELIN E 

Name 

Auto Mechanics RP. 27/1953 

Biology RP 50/1963 

Chemistry RP, 51/1965 

El e merits of Construct ion 
Technology Senior Div. 1970 

English Inte rme d i a t e Div. 1969 

Francais, Niveau Intermediarie, 1971 

Man and the Earth, Intermedi ate 
Division 1973 

German, Intermediate and Senior 
Division 1961 

Man, Science and Technology 1972 

Science Intermediate Division 
Interim Revi s i on 1972 

Screen Education 1970 

Typewriting Personal I & S 31A 1969 

World Religions 1972 

I nf ormat i cs I ntermedi ate and 
Seni or Di vi si ons 197 2 

Mathemat i cs Senior Division 1 97 2 

Mus i c , I ntermadi ate Division 
Interi m Revi si on 1 972 

People and Politics Senior 
Division 1972 

P. 1 .J. 1. 1 972 



TABLE 2. CURRICULUM GUIDELINES - PRIMARY EVALUATION 
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IV • MULTI-YEAR OPERATIONS MANUA L 

On the basis of the field test and the primary 
evaluation, each phase of an evaluation was formalized 
and documented in a two volume document entitled 
Curriculum Guideline Implementation Evaluation Operations 
Manual , 

This document has been designed to be a step-by- 
step guide for replicating future evaluations of guidelines. 
The manual consists of two volumes; the first containing 
a detailed discussion of each step in an evaluation and 
the second a technical supplement to be used as a reference 
with the first. The following section summarizes each 
section of the manual. 

V. SUMMARY OF PART I 



1 • Preliminary Decisions - Section A 

Section A described the initial decision-making 
process and the broad policy decisions that must be made 
before undertaking an evaluation. This phase involves the 
initial conceptual planning that will determine the scope 
of the evaluation. Decisions must be made concerning the 
selection of guidelines to be evaluated and regions and 
personnel to be accessed. 

2 . Population Det ermiiiation - Section B 

Section B describes the first operational phase 
in the evaluation sequence - the task of calculating the 
population available to respond to each guideline under 
evaluation. Forms are provided to aid in the organization 
of this phase. 

O 
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3 . Sample Size Determination - Section C 

Section C describes the methods of calculating 
the sample sizes required for the survey. These calcula- 
tions are based on the determination of the populations of 
Section B. A table of sample sizes and questionnaires 
required is provided to facilitate this process, and 
planning forms are provided as organization aids. 

4 . Preliminary Activities in Preparation for 
Printing - Section D 

Section D describes those activities that must be 
completed before initiating the printing of any material. 
A number of minor activities are required before the actual 
printing of the questionnaires. The material consists of 
questionnaires , letters of instruction , reminder notices , 
etc . 

5 . Printing - Section E 

Sect ion E describes the acti vi t ies involved in 
printing all the material for each personnel level to be 
surveyed. The section is primarily concerned with tabu- 
lating the numbers of each type of material required. Forms 
are included to facilitate these calculations. 

6 . Mailing Activities - Section F 

Section F is intended as a self-supporting manual 
to accomplish the complicated collating activities required 
to survey each personnel level for any one guideline . 
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7 . Activities Associated with the Return of the 
Questionnaires - Section G 

Section G describes the activities associated 
with the return of the questionnaires before data 
processing can begin. This section includes detailed 
instructions regarding questionnaire pre-editing as well 
as instructions to accompany the questionnaires when they 
are keypunched. 

8 . Data Pre proces sing - Section H 

Section H presents a non-technical description of 
the computer programs used in the two major data pre- 
processing activities. A technical supplement to Section H 
is included in Part II. 

9 . Requests for Analyses - Section I 

Section I describes the kinds of analyses which may 
be requested and tiie methods by which the requests may 
be made and processed. The technical information 
supporting the information in this section is documented 
in Part II. 

10 . Analytical Methodology - Section J 

Section J describes the detailed analyses available 
from both standard reports and from SPSS. The first part 
describes the considerations which must be made to perform 
any analysis. The second part describes the specific items 
which constitute the bases of the primary, secondary and 
other analyses. 
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VI. SUMMARY OF PART II 

1, Guideline Codes - Section A 

Section A describes the uses of guideline codes 
in evaluations and includes a list of all guidelines 
available as of January 1974* (Guideline codes are 
internal Curriculum Guideline reference codes,) 

2 • List Programs - Section B 

Section B is the techni cal supplement to Sections 
D and H of Part I, This section describes the computer 
programs used to generate the list of subjects taught in 
each secondary school, sorted by board and region. Included 
in this section are forms which must be completed in order 
to obtain these lists with appropriate subject (guideline) 
information for any evaluation. These lists are used to 
determine the available populations for a survey, 

3 , Data Pre proces sing Programs - Section C 

Section C describes the technical specifications 
of the questionnaire preprocessing programs. The data 
edit/reformatting programs are discussed in terms of the 
input requirements and file formats, 

4 , SPSS Specifications - Section D 

Section D describes the technical specifications 
of the SPSS programs and represents a technical users 
supplement to Part I Section J, 

5 . Reporting Gt ructures - Section E 
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Section E contains the Report Description Manual 
for the standard reports shown in Appendix B, The Section 
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is intended to be a self-supporting section which can be 
easily removed and duplicated for distribution to those 
personnel who receive the Standard Reports, 

6 • Set of All Forins - Section F 

Section F contains a complete set of all the forms 
contained in the main body of the operations manual. The 
purpose of the section is the provision of these organi- 
zational aids for duplication at the commencement of an 
evaluation . 

VII. CONCLUSION 

This paper summarizes the activities which have 
taken place in the Curriculum Guideline Implementation 
Evaluation Pro j ect . 

The primary objective of the project was the design 
of a system to permit Ministry of Education personnel to 
monitor changes in the effect of curriculum guidelines 
in Ontario. An integral part of this system is a computer 
based inf ormat ion system consisting of data collected at 
selected times from educational personnel in the provi.nce. 
In order to ensure consistency in the data col lection 
methodology a multi-year operations manual was designed 
describing in detail each operation required to replicate 
the data collection. 
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FINAL QUESTIONNAIRE 
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APPENDIX C 

REGIONAL RESPONSE BY GUIDELINE 
BY PERSONNEL LEVEL 
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1 3. B 
7 .g 
1 , 0 



61 
1 6.6 
6.2 
0 .6 



87 
22.9 
6.6 

0 . e 



4B 

12.3 
3 . 7 



27.1 
3 .4 



0 

0. 0 
0.0 
0.0 



2 5 
6. 9 
3.2 
0. 2 



1 

I . 7 
0 . I 
0 .0 



3 3 



4 . 2 
0,3 



9 

7.6 
1 .2 
0. 1 



I 5 
7 . I 
I . 9 
0 . I 



62 
3.2 



PtcirN 10 now 

TOTAL 



34 

7 . 0 
4 , 4 
0 . 3 



21 
5.5 
2 . 7 
0 . 2 



3 6 
9.2 
4 .6 
0. 3 



3C0 
7. I 
3 9.5 
2 . V 



707 



\?f'2 
1 .2. 2 



1132 
10.fi 



1 225 
11.7 



I 501 

14.3 



I 31 2 
12.5 



700 



3. 0 
0. 1 
0 . 0 



35 
9. 7 
2 .9 
0. 3 



3 

5.0 
0.2 
0 . 0 



63 
1 ?.) 
5.2 
0.6 



1 7 
14.3 

1 .4 

0 . 2 



I 7 
8.1 

I .4 

0.2 



. 90 
11.9 
7 . 4 
0.9 



40 
8.2 
3.3 

0.4 



39 
10.3 
3.2 

0 . 4 



44 

11.3 

3.6 
0 . 4 



6 03 
13.9 
4 9.4 

5.8 



I 220 
11.6 



460 

4 • 4 



51 6 

4 « 9 



412 

3.9 



3 1 6 
3.0 



60 3 
7 . 7 



66 
0«& . 



260 . 
2*5 



33 
0. 3 



362 
3.5 



to. 

0 . 6 



522 
5 . 0 



I 1 9 
1 . 1 



21 0 
. 2.0_ 



755. 
7.2 



4 69 
4 . 7 



380 
3.6„ 



391 . 
3 . 7 



4 323 

4 1.3 



1 0477 
100.0 



REGic:iAL svyyMY or respokse-.. , 

BY GUIDELIKi: 



:ERLC 



